Variational autoencoders (VAEs) are widely used deep generative models capable of learning unsupervised latent representations of data. Such representations are often difficult to interpret or control. We consider the problem of unsupervised learning of features correlated to specific labels in a dataset. We propose a VAE-based generative model which we show is capable of extracting features correlated to binary labels in the data and structuring it in a latent subspace which is easy to interpret. Our model, the Conditional Subspace VAE (CSVAE), uses mutual information minimization to learn a low-dimensional latent subspace associated with each label that can easily be inspected and independently manipulated. We demonstrate the utility of the learned representations for attribute manipulation tasks on both the Toronto Face and CelebA datasets.

Updated (due to rebuttal & discussion w/ R2): The authors reiterate in their rebuttal their core contributions of "extracting information beyond binary labels" and "attribute manipulation from a single image", together with the promise to clarify it in the paper. The contributions are relevant to the community, since this form of hierarchical disentangling seems novel. That said, there is some degree of similarity of the proposed variational approach to IGN (Deep Convolutional Inverse Graphics Network https://arxiv.org/abs/1503.03167). IGN is cited, but not discussed in detail, and an empirical comparison is not provided, despite being applicable to the current setting as well. Nevertheless, since the selling point of the paper seems to be the ability to discover sub-categories from only category labels, which is not addressed in IGN and is an interesting empirical find, I increased my score to be marginally above the acceptance threshold.

Variational autoencoders (VAEs) are widely used deep generative models capable of learning unsupervised latent representations of data. Such representations are often difficult to interpret or control. We consider the problem of unsupervised learning of features correlated to specific labels in a dataset. We propose a VAE-based generative model which we show is capable of extracting features correlated to binary labels in the data and structuring it in a latent subspace which is easy to interpret. Our model, the Conditional Subspace VAE (CSVAE), uses mutual information minimization to learn a low-dimensional latent subspace associated with each label that can easily be inspected and independently manipulated.